An Effective Machine Learning-Based Garbage Data Filtering Algorithm for Social Networking Services (SNS) Big Data Processing

Abstract

Social networking services (SNS) such as Facebook, X (formerly Twitter), Instagram, and Reddit generate enormous volumes of data every second. A substantial portion of this data is “garbage”: spam, duplicate content, fake news, bot-generated posts, advertisements, irrelevant comments, and noisy text. This study proposes a machine learning-based garbage data filtering algorithm tailored to SNS platforms. The approach combines a robust preprocessing pipeline — tokenization, stop-word removal, and both lexical and contextual feature extraction — with classification models ranging from traditional supervised classifiers such as Random Forest to advanced deep learning models such as BERT, enabling accurate discrimination between valuable and garbage content. Feature engineering incorporates text length, repetition metrics, spam indicators, and semantic embeddings to support fine-grained content discrimination. The method is evaluated on publicly available SNS datasets, including Twitter and Reddit posts as well as manually labeled samples, using precision, recall, F1-score, and robustness across multiple domains and topics as performance measures. The proposed algorithm consistently outperforms baseline methods in filtering garbage data while preserving relevant information, demonstrating its practical value for real-world deployment. The system is further designed to support real-time or near real-time processing through scalable architectures such as Apache Spark and Apache Kafka, allowing seamless integration with existing big data pipelines and making it suitable for large-scale deployment in SNS monitoring tools, trend-analysis systems, and recommender engines.

Introduction

This paper presents an Effective Garbage Data Filtering Algorithm for Social Networking Service (SNS) Big Data Processing using machine learning and deep learning techniques. Social media platforms such as Twitter, Facebook, and Instagram generate massive amounts of user-generated content, but a significant portion of this data consists of spam, duplicate posts, bot-generated messages, irrelevant content, and other forms of low-quality information. Such garbage data increases storage and processing costs while reducing the accuracy of analytics and machine learning models.

Traditional filtering techniques based on rules and keyword lists are insufficient because social media language and user behaviour constantly evolve. The proposed system addresses this problem through a combination of data preprocessing, semantic feature extraction, and intelligent classification. It uses both statistical and contextual features to distinguish useful content from irrelevant data.

The methodology includes:

• Data collection from SNS platforms using APIs such as Tweepy and PRAW, supporting both real-time and batch processing.
• Data preprocessing involving text normalization, tokenization, stop-word removal, lemmatization, URL removal, and language detection.
• Feature extraction using TF-IDF, n-grams, Word2Vec/BERT embeddings, and additional indicators such as text length, hashtag frequency, link usage, and character repetition.
• Classification using a hybrid approach combining Random Forest and BERT, with additional evaluation of models such as XGBoost and Decision Tree.
• Deployment and scalability through tools such as Apache Kafka, Apache Spark, MongoDB, Flask, and Docker for large-scale real-time filtering.

The system architecture is designed for scalable SNS data processing, enabling real-time monitoring and efficient filtering of large data streams. Transformer-based models such as BERT provide better contextual understanding and higher classification accuracy, while traditional machine-learning models offer faster processing for real-time applications.

The study compares different algorithms using evaluation metrics including precision, recall, F1-score, accuracy, and processing time. Results show that advanced models such as BERT, Random Forest, XGBoost, and Decision Tree outperform the baseline Naïve Bayes approach, achieving improved accuracy in garbage data detection.

Conclusion

The proposed system successfully addresses the challenge of garbage data in SNS streams by introducing a machine learning-based approach that intelligently filters such data, thereby enhancing the overall value of social media analytics. [8] Throughout this study, the shortcomings of existing garbage data filtering systems were identified and analysed. Rule-based and keyword-centric systems, while simple, fail to adapt to the rapidly changing language and patterns used on social media; such systems are prone to high false-positive rates, limited scalability, and a lack of contextual understanding — issues that the proposed solution aims to overcome through more advanced and adaptive techniques. The proposed machine learning-based garbage data filtering algorithm effectively improves the quality of SNS big data by accurately identifying and removing spam, duplicate, irrelevant, and noisy content. By filtering out unwanted data before analysis, the system improves processing speed, reduces storage and computational costs, and increases the accuracy of downstream analytics. Its scalable, automated design makes it suitable for large-scale social networking platforms, supporting reliable insights and better decision-making in big data environments.

References

[1] Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT. [2] Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780. [3] Kumar, A., & Sebastian, T. (2020). Spam detection on social media using machine learning techniques. International Journal of Computer Applications, 177(38), 1–6. [4] Gulli, A., & Pal, S. (2017). Deep learning with Keras. Packt Publishing Ltd. [5] Rajalakshmi, K., & Krishnamurthi, R. (2019). Big data preprocessing techniques for social media data. In Proceedings of the 3rd International Conference on Computing Methodologies and Communication (ICCMC), IEEE. [6] Aggarwal, C. C. (2018). Machine learning for text. Springer. [7] Kowsari, K., Heidarysafa, M., Brown, D. E., Jafari Meimandi, K., & Barnes, L. E. (2019). Text classification algorithms: A survey. Information, 10(4), 150. [8] Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. [9] Kumar, K. T. K. (2025). A machine learning framework for cyber bullying and hate speech detection on social media. International Journal of Research and Analytical Reviews, 12(2). http://ijrar.org/viewfull.php?&p_id=IJRAR25B3584 [10] Chen, T., & Guestrin, C. (2016). XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 785–794. [11] Kumar, K. T. K. (2022). Analyzing app rating using natural language processing and machine learning. International Research Journal of Engineering and Technology (IRJET), 9(5). https://irjet.net/archives/v9/i5/irjet-v9i5531.pdf [12] Zhang, Y., & Wallace, B. C. (2017). A sensitivity analysis of (and practitioners\' guide to) convolutional neural networks for sentence classification. In Proceedings of IJCNLP. [13] Breiman, L. (2001). Random forests. Machine Learning, 45(1), 5–32. [14] Quinlan, J. R. (1986). Induction of decision trees. Machine Learning, 1(1), 81–106. [15] Zaharia, M., Chowdhury, M., Franklin, M. J., Shenker, S., & Stoica, I. (2010). Spark: Cluster computing with working sets. In Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing (HotCloud). [16] Kreps, J., Narkhede, N., & Rao, J. (2011). Kafka: A distributed messaging system for log processing. In Proceedings of the NetDB Workshop. [17] Shu, K., Sliva, A., Wang, S., Tang, J., & Liu, H. (2017). Fake news detection on social media: A data mining perspective. ACM SIGKDD Explorations Newsletter, 19(1), 22–36. [18] Kumar, K. T. K. (2023). Automatic text summarization (ATS) by using recursive neural networks. International Journal of Creative Research Thoughts, 11(4). http://www.ijcrt.org/papers/IJCRT2304392.pdf [19] Ferrara, E., Varol, O., Davis, C., Menczer, F., & Flammini, A. (2016). The rise of social bots. Communications of the ACM, 59(7), 96–104. [20] Pennington, J., Socher, R., & Manning, C. D. (2014). GloVe: Global vectors for word representation. In Proceedings of EMNLP.

Copyright

Copyright © 2026 K. Tulasi Krishna Kumar, E. Vinay. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.